1.1 Announcements and Due Dates

• Don’t forget about the reading evidence - due only 1 week from today!

• Continue work on the article critique and research proposal - we are now into the final stretch. Use the time in class to help cut down on the amount you have to do at home! (But still do work at home on these).

• I will have office hours again at 3:00pm - 6:00pm EST in AuSable 1307 on Friday 11/22/2024. Please consider coming if you have recently been struggling or would like to discuss the projects. I am available via email as well.

1.2 Last Unit Content

• We talked more about the many difficult pitfalls of internal validity in experimental designs, and how to reduce them!

• We discussed how to cope with null results and possible causes for this result.

• We talked about narratives and details in writing and critiquing research.

2.1 Areas for Review

• The Example: I want to explore the relationship between academic success and participation in a new support program, as I believe the program should result in a higher GPA. I recruit from the population of students on the honors track, who are all enrolled in the same general education classes for the next two semesters. I put up a poster with a QR code, asking for participants in the study, in order to recruit people. With my chosen participants, I randomly assign students into two separate groups, whether or not they participate in the program. Then, I record all students’ current GPA. I then start the new support program for half of the participants, and for those who are not in it, I do not do anything. After two semesters, I stop the program and re-measure all of the GPAs. I conclude that there is no difference in GPA between the students who do participate in the program, and those who do not. Throughout the study, I ensure that none of the students partake in other support programs, like tutoring or mentorship.

• Putting up a poster asking for participants is effectively a form of convenience and self-sampling \(\rightarrow\) non-probabilistic sampling (as it is not based in randomness) \(\rightarrow\) low generalizability

• (Everyone got points) Counter balancing of conditions is only necessary on within-subjects designs, as those are the type that are affects by order effects (i.e., exposure to one condition impacts the next one).

  • The above example is a between-groups design
  • Random assignment to groups is for between subjects designs to control for selection effects.

3.1 Quiz Content

• Covers all content from 11/12 class meeting, including but not limited to:
  • Chapter 11 of Morling Textbook
  • Lecture on Chapter 11
• Any last minute questions?

3.2 Quiz Rules

• From the Syllabus:
  • Each quiz is 10 multiple-choice questions, 1 point for each question
  • Quizzes will be taken at the start of the class period on the Blackboard LMS
  • Quizzes will be on content covered in the previous lecture and the associated reading for that lecture
  • Quizzes are timed, 23 minutes only (previously was 15 minutes). If you finish before time is up, please remain in class and find another activity to work on quietly
  • Quizzes are open-note and open-book, that is, you are allowed to use those resources during the quizzes. Thus, they reward good structure in thoughtfulness in your notes and preparation
  • You may not collaborate with others during the quizzes, or discuss questions with other students after the quiz. You cannot use AI tools or the internet to help you during the quiz
  • Quizzes and exam will be ended early if all students are clearly finished and content with their answers
  • Quizzes will be graded promptly and reviewed the following week

4.1 Textbook Objectives

• Explain why researchers combine independent variables in a factorial design.

• Describe an interaction effect in both everyday terms and arithmetic terms.

• Identify and interpret the main effects and interactions from a factorial design.

4.2 Professor’s Objectives

• Understand and produce examples of when factorial design would be useful and/or appropriate

• Discuss some basic statistics procedures that can be used with these designs

5.1 Chapter Overview

• Up until now, we have only talked about experimental designs that deal with one manipulated/independent variable and one measured/dependent variable.

• However, we have designs that can look at two (or more) IVs at once and see their individual and combined impact on the DV!

• We refer to these as factorial designs.

6.1 Overview

• We can add a second (and third) independent variable if we are curious about more than one.

• In addition to the individual effects of both of the IVs, we also get an interaction effect that describes how they change each other’s relationship with the outcome.

• Statistically, we might say this interaction is a “difference in differences”

  • Practically, this means that the differences between our groups may be different based on some other trait.
  • More on this later

6.2 Intuitive Interactions

• When confronted with a causal relationship, sometimes we might say, “well it depends” - what it depends on is the second (or third) IV

• We can see this even in our personal experiences, and many relationships do depend on other factors

• Example: I am assessing how spicy I like my food (on a scale of 1 to 10; my outcome). First, is it cold or hot outside (IV 1)? Second, am I eating Thai or Italian (IV 2)? It is possible that my answer will be different based upon both of the IVs.

• 4 Possible Outcomes:

• I like all of my food spicier when it is hot - Weather effect, but not food

• I like Thai food spicier that Italian, regardless of weather - Food effect, but not temperature

• Whether I like by food spicy or not depends on both the weather, and type of food - interaction effect

• Specifically, we are looking to see whether we have a crossover interaction, like in the graph below:

• My preference for spice doesn’t change, regardless of food type or weather - null findings

6.3 Study Two IVs

• When we work with more than one IV, we use a factorial design.

• This creates more outcome unique conditions = # of Conditions in IV 1 x # of Conditions in IV 2 = total number of conditions

• Both IVs do not have to be manipulated. Often, one will be some categorical, measured trait (e.g., gender, ethnicity, etc.)

• In addition to our statistics, we should show these differences in plots! Interaction effects become especially clear with visual evidence.

6.4 Limit Testing

• Factorial designs can help us find whether outcomes are different for different types of people.

• A strong intervention may not be as effective in a different group of people.

• This can be a boon to our external validity, as we demonstrate findings in a more heterogeneous group.

• We also can establish whether one variable appears to moderate another on the relationship with the outcome variable.

6.5 Test Theories

• For some theoretical reasons, we may have good reason to believe that an effect differs based on some demographic variable.

• Example: I have a new intervention meant to encourage flexibility in learning and taking in new content. However, I recognize that the neuroplasticity of older adults is just lesser in general. Therefore, I believe my intervention will likely be more effective for younger adults, than it will for older adults.

• In essence, we may be able to add nuance and “it depends” to our hypotheses and investigate with factorial designs.

6.6 Main Effects & Interactions

• Main Effects are those that come from each IV on the outcome.
  • The main effect is calculated as an average over the levels of the other IV. Similar to how we “control” for other variable in multiple regression.
  • You have 1 main effect for each IV
• Marginal Means are the statistic that we use to determine whether a main effect is present
  • We can test significance by taking the difference of the two marginal means, and calculating 95% CIs. If CIs contain 0 \(\rightarrow\) non-significant

• An interaction effect can be detected by looking at the differences of the main effect differences. If they are significantly different from one another, then we would say that there is an interaction effect
  • Interactions are often treated as more important, theoretically, that main effects - when they are significant.
• Conventional wisdom: If interaction is significant, focus on that mostly. If interaction is non-significant, focus on main effects of IVs.
  • Interpreting the main effects with a significant interaction can be leaving out important information!
• Stats sidebar: This type of analysis is usually done via Two-way ANOVA, which does all the work of calculating significance of interactions, and main effects for us.

7.1 Overview

• Just like with other experiments, we can lay out a factorial design as being between-groups or within-groups.

• But, we can designate each variable as between or within, leading to a total of 3 possible designs:

  • Independent-Groups Factorial
  • Within-Groups Factorial
  • Mixed Factorial

7.2 Independent-Groups Design

• This is when all IVs are between-groups (i.e., participants are arranged into entirely separate groups)

• One nuance is that this will likely require the largest sample size, as each group will have about 1/4th the total number of participants

7.3 Within-Groups Design

• Much like with previous within-groups designs, this is when participants see every possible condition.

• One thing to watch out for is the need for counterbalancing to prevent order effects

  • Think about how many permutations of condition orders you may need!

7.4 Mixed Factorial Design

• This is when one IV is between-groups, and the other is within-group.

• This is fairly common if we have one demographic variable (between-groups) and one manipulated variable that both demographics are exposed to each level (within-groups).

7.5 More Conditions

• Many IVs are going to naturally have more than one level
  • E.g., race, ethnicity, gender, etc.
• We can use these in factorial designs all the same - and we write it as: \(A x B\) Design.
  • Where \(A\) = Number of conditions in IV 1
  • Where \(B\) = Number of conditions in IV 2
• Statistics here get more complex to interpret - but a good starting point is to use a line plot just like what we have done previously and see if lines cross or are parallel.

7.6 More IVs

• Prof. Paul Moes: “God himself cannot interpret a 4-way interaction - neither can you”

• We can do 3 IVs, but with each additional variable the interpretation becomes exponentially more difficult and complicated.

  • One popular alternative is to do this as a multiple regression model instead
  • Stat sidebar: ANOVA and linear regression are both types of the general linear model, so, in a roundabout way, these are actually equivalent!

• Remember to think carefully about what sorts of conclusions you can draw with a design before you use it, and whether an alternative provides a more parsimonious conclusion.

8.1 Reading Empirical Articles

• Look for words like …
  • “Two-way ANOVA”
  • “Factorial”
  • “Interaction” or “Main Effects”
• You may also see phrasing like “2 x 2 design”, referring to the two conditions of each IV.

8.2 In Popular Media

• Look for words like …
  • “depends”
  • “Only when”
• You may also look for demographic variables …
  • “For males this was the results, but for females…”

Week 12 Lecture - Complex Experiments || Undergraduate Research Methods in Psychology